An algorithm for gesture detection and tracking in HCI (human-computer interaction) is designed to meet real-time and accuracy requirements. An innovative conception, which includes using distinguishing detection methods to detect hand-gesture for different conditions and using searching-box to decrease the searching zone, is proposed. The result shows that the detection rate can reach 97.36% while the missing rate lowers than 5%. It is robust to various external factors. However, it also meets the real-time, as the frame rate can reach 19.42. Compared with TLD, this algorithm has not only magnitude improvement in processing speed but also obvious advantages in accuracy.